{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Tutorial 01: Random Maze Walk",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "swift",
      "display_name": "Swift"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5qXhi3slajqW",
        "colab_type": "text"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2eCrLoRYbJuV",
        "colab_type": "text"
      },
      "source": [
        "##### Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1L-ZwYrwbCeZ",
        "colab_type": "text"
      },
      "source": [
        "```\n",
        "// Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "// you may not use this file except in compliance with the License.\n",
        "// You may obtain a copy of the License at\n",
        "//\n",
        "// https://www.apache.org/licenses/LICENSE-2.0\n",
        "//\n",
        "// Unless required by applicable law or agreed to in writing, software\n",
        "// distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "// See the License for the specific language governing permissions and\n",
        "// limitations under the License.\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U_g59bfqbUPy",
        "colab_type": "text"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/examples/blob/master/community/en/swift/open_spiel/T01_RandomMazeWalk.ipynb \"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/examples/blob/master/community/en/swift/open_spiel/T01_RandomMazeWalk.ipynb \"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KoLzyHV2JWtw",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "# Introduction\n",
        "\n",
        "This Colab will use the maze below, and have the Robot walk randomly until it either falls into a hole or reaches goal.\n",
        "![ExampleImage](https://github.com/deepmind/open_spiel/blob/master/swift/Examples/GridMaze/Images/GridMazeExample.png?raw=true)\n",
        "\n",
        "For a detailed description of how this maze or how the GridMaze Environment works in general, please see the [Introduction to GridMaze Environment](https://colab.research.google.com/github/tensorflow/examples/blob/master/community/en/swift/open_spiel/Introduction_to_GridMaze_Environment.ipynb) Colab.\n",
        "\n",
        "## Installing OpenSpiel and imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sSui_vb7LIV1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%install '.package(url: \"https://github.com/deepmind/open_spiel\", .branch(\"master\"))' OpenSpiel\n",
        "import TensorFlow\n",
        "import OpenSpiel"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ObZCC6zsLElE",
        "colab_type": "text"
      },
      "source": [
        "## Creating the GridMaze Environment\n",
        "The following code creates the maze environment depicted above."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kZRlD4utdPuX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "var maze = GridMaze(rowCount: 6, columnCount: 6)\n",
        "maze[1, 1] = GridCell.start(reward: -1.0)\n",
        "maze[2, 2] = GridCell.hole(reward: -100)\n",
        "maze[2, 3] = GridCell.space(reward: -2.0)\n",
        "maze[3, 2] = GridCell.space(reward: -2.0)\n",
        "maze[3, 4] = GridCell.hole(reward: -100)\n",
        "maze[4, 0] = GridCell.space(reward: -1.0)\n",
        "maze[4, 2] = GridCell.bounce\n",
        "maze[4, 4] = GridCell.goal(reward: 1.0)\n",
        "maze[4, 5] = GridCell.space(reward: -1.0)\n",
        "maze[2, 4].entryJumpProbabilities = [(.Relative(1, 0), 0.5), (.Welcome, 0.5)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wpGeqhR3L0gI",
        "colab_type": "text"
      },
      "source": [
        "## Printing the Maze\n",
        "The maze environment has now been created, it can be displayed by calling `printMazeAndTable`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b-ZFX8K5OJ6H",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "maze.printMazeAndTable(header: \"Maze Environment\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2fXEr0VvRl25",
        "colab_type": "text"
      },
      "source": [
        "## Random Walk Loop\n",
        "\n",
        "The below code make the Robot walk around randomly in the maze. In this process, it will eventually reach one of two terminal states:\n",
        "* Hole\n",
        "* Goal\n",
        "\n",
        "The main iteration loop will continue until the Robot reaches goal. In other words, if it falls into a hole, the loop will start over until it reaches the goal state."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gjSTGj1USE8U",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "var gamesPlayed = 0\n",
        "var continuePlaying = true\n",
        "while continuePlaying {\n",
        "  gamesPlayed += 1\n",
        "  var gameState = maze.initialState\n",
        "  var actionSequence = [String]()\n",
        "  \n",
        "  // Let the Robot take actions (walk betwen cells) until it either reaches GOAL\n",
        "  // or falls into a HOAL (those are the only two terminal cells)\n",
        "  while !gameState.isTerminal {\n",
        "    \n",
        "    // The current informationState, denoting the cell position the Robot is\n",
        "    // currently at. If you create a learning algorithm, this will be the key\n",
        "    // to v-/q-/policy-tables: \n",
        "    //    let currentInformationState = gameState.informationStateString()\n",
        "    // This will be extensively used in learning algorithms in later tutorials.\n",
        "    \n",
        "    // Select a random action from the legal ones in this state\n",
        "    // The GridMaze.Action enum has members: .LEFT, .UP, .DOWN, .RIGHT\n",
        "    // Since some cells cannot be even attempted to be entered (e.g. WALL), all\n",
        "    // gameStates may not return all four members\n",
        "    let actionIndex = Int.random(in: 0..<gameState.legalActions.count)\n",
        "    let actionToTake = gameState.legalActions[actionIndex]\n",
        "    \n",
        "    // We now have an actionToTake from current informationState\n",
        "    // Let's have Robot move in that direction!\n",
        "    gameState.apply(actionToTake)\n",
        "    \n",
        "    // Store the action taken to print that later\n",
        "    actionSequence.append(actionToTake.description)\n",
        "    \n",
        "    // If Robot made it to GOAL then we're done\n",
        "    if gameState.isGoal {\n",
        "      print(\"*** AWESOME. Robot made it to Goal at game: \\(gamesPlayed) with reward/cost: \\(gameState.utility(for: .player(0)))\")\n",
        "      print(\"    Sequence of actions used to solve the maze: \\(actionSequence)\")\n",
        "      continuePlaying = false\n",
        "    } else if gameState.isTerminal {\n",
        "      print(\"Robot failed to solve the maze at game: \\(gamesPlayed) with reward/cost: \\(gameState.utility(for: .player(0)))\")\n",
        "    }\n",
        "  }\n",
        "}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R1v6Eqe_WqaJ",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "## Summary\n",
        "This tutorial demonstrated the Robot talking random actions to eventually reach the goal state. [Later tutorials](https://github.com/tensorflow/examples/blob/master/community/en/swift/open_spiel) will cover more advanced examples, for example how to create generic algorithms the Robot will use to optimally\n",
        "find a solution to any maze problem (or other environment problems for that matter).\n",
        "\n",
        "## Join the community!\n",
        "If you have any questions about Swift for TensorFlow, Swift in OpenSpiel, or would like to share your work or research with the community, please join our mailing list [`swift@tensorflow.org`](https://groups.google.com/a/tensorflow.org/forum/#!forum/swift)."
      ]
    }
  ]
}